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NetScope: Traffic Engineering for IP Networks 



Anja FHtlma-TiTi. All»erl GreenUerg. Carslen Lmvl. Xirk Reingold. an«l Jennifer Re\for<l 

Xelworking a.n<l Di^^lrilniied Systems 
ATl^'T TjaLs - Researrli 
{an j a , albert , lund , reingold , j rex}®research . att . com 

Afrinaging large IP networks requires an nrnlerpii.an<ling of i.lie oirreTii. (rafTir flows, roiiuing poliries, ami 
nei.work ronfigiirai.ion. Yei , I lie s( aie-of-llie-arl for managing IP nei.wwks involves manual ronfigiirai ion of 
eadi IP router, antl iraflir engineering liasetl on limitetl mea<5iiremenus. Tlie networking intliisirj* is sorely 
larking in software sysiems ilial a large Iniemei. Ser\ ire Provider (1SP| ran nse io support. i.rafTir mea- 
snremenl antl network motleling, Uie untlerpinnings orefTerllve t.rafTir engineering. Tliis paper (lesrrihes lite 
ATX.T T;al>s XpfSmpf^ a nnifietl set. ofsoHware tools for managing I lie perform an re of IP 1»arkltone networks. 
Tlie key i<lea heliintl Xet.Srope is t.o generate gloKal views of t.lie network, on llie ba^^is of ronftgnrat ion and 
usage tlat.a assoriate<l wlllt (lie intlivi<lnal network elements. Having rreat.eil an appropriate gloKal view, 
we are al»le t.o infer and visuali7e the net work-wi<le impliralions of loral rlianges in Iraffir, ron figurat ion, 
antl rontrol. T'sing XeiSrope, a network provider ran experiment, willi rlianges in network ronfignralion 
in a simiilat.e<l environment, rat.lier than t.lie operational network. In addition, tlie tool provi<les a sonn<l 
framework for additional motlnles for net.work 0pt.imi7at.i0n antl performanre <lel nigging. We demonstrate 
flte rapaltilit.ies of tlie t.ool tlirongli an example trafTir-engineering exerrise of lorat ing a lieavily-loade<l link, 
itlenlifying wliirli iralTir <leman<ls flow on t.lie link, an<l rlianging tlie ronftgiirat ion of intra-tlomain routing 
Io rethire tlte rongestion. 

1 Introduction 

Traffic f^iigin*^ring aims to opTimiz^ n^^twork: performanre rlirongh thr^. mtt^.^T^tf^ artivit.ies - measuring t.raffir. 
modeling t.h*^ network, and sele<-ting merhaiiismsfor rontrol ling the trafRr. Unfort.nnatdy, large Tnt^^rnet Ser\nce 
Providers (TSPs) have few software systems and tools to support traffir measurement and network modeling, the 
underpinnings of effe^-tive traffic engineering. Seemingly simple qnestions ahont topology, traffic, and routing 
are surprisingly hard to answer in today s TP networks. A tremendous amount of work has gone into developing 
me<".hanisms and proto^^'ols for control ling traffic. Tndeeii. most of the work in the Internet Engineering Task Force 
(TETF) concerns the aspect of traffic engHneering concerned with traffic control. By comparison, little work has 
gone to supp->rt traffic measurement and network m«">deling in operational networks. (Notahle exceptions in the 
TKTF include the TPPM and RTFM working groups.) Unless control mechanisms are driven by the appropriate 
measurements and understanding from well-testei-i models, the f.euefit of the controls will l ie limited. 

This paper describes the AT&T Tial -s WAS'copf^. a unified set of s<'>ftw-are tools for traffic engineering in 
operational TP T>acklx>ne networks. The key idea behind XetScope is to generate glol ial views of the network, 
on the Tiasis of configuration and usage data associated with the indi\'idual network element^. ITavnng created 
an appropriate global view, we are able to infer an<T visualise the network-wide implications of local changes 
in traffic, configuration, and control. NetS<"ope provides an extensible and powerful platform for '*what.-if'' 
traffic-engineering investigations. Using XetScope. a netw-ork pro\'ider can experiment with changes in network 
configuration in a simulateiT environment, rather than in the operational network. Tn arldition. the tool provides 
a sound framework for additional modules for network optimisation and performance del .uggiug. 

Tn order to illustrate some of the capabilities of XetScope. we focus on one compelling application: the 
prolilem of configuring intra-domain routing basefl on the underlying network topology and traffic .Temands. This 
application drives the choice of an appropriate top<~>logical view, as w^ell as the methods for traffic aggregation 
and routing. The task can >>e decomposed into several steps, each drawing heavily on the underlying modules 
and the flexible visual i;^ation environment. By joining topology and usage data. XetScope displays the utili;^ation 
of each of the links, and identifies the most heavily-loade^T link. Then. NetS^^opes integrated view of traffic. 



topology. anH rniiTiiig enal iles lis to id^=:ntify whirli soiirr^vsink pairs are responsild^ for tlie ^^OTiijj^tion. Bas^i 
on til is information. w*=^ change th^ oonfign ration of intra-domain routing; (by r.1iano;ing an OSPF link weight) to 
direct some, of these traffic demands away from the congested link. KetS^^'ope then re<-ompiit.PS the routes and 
the resnlting 1oa«-| on each hnk. provi<iing an estimate of how the traffic would flow after the change. 

1.1 Network Engineering Constraints 

Before plnnging into details, let iis first consider the factors that are driving; the neeii for better network 
engineering tools: 

♦ Service quality: Increasingly, customers are demanding more strin.y;ent assurances M-ith regar< ls to perfor- 
mance, relial iility. and secnrity, in the form of Service T.evel Agreements (ST^As). ([Instomers are developing 
certification and ongoing testing procednres. to as.snre compliance to these SLAs. Applications are emerg- 
ing, snch as TP Voice, that hy their nature rerpiire high quality data transport, as measured hy delay, loss, 
and jitter. Since TP networks do not have explicit support for performance guarauteps. network operators 
must carefully coordinate the operation of the indi\-idual network elements to realise customer ST. As. 

♦ Interdependent tunable parameters: At pr^^nt, vendors of netw^ork components provide TS Ps with 
little or no low-level control over the T»asic mechanisms respousil'»le for packet s*"heduling. TtufFer manay;e- 
meut. and path selection. Instead, hackhone pro\Hders are force«T to understand a large set of inter-related 
parameters, each to s<">me degree affe<^ting confitj;uration and operation. To date, an TSP must manage 
\fs Kacklx>ne network, and complicate^T peering relationships with neighboring providers, by tuning these 
knol?=i through a comliinatiou of intuition, testing, and trial-and-error. 

♦ Network growth: Tn<Tividual l->ackl-tone networks are growing rapidly in size, speed, and scope, and the 
industry is consolidating many disparate networks into larger integrated networks. As a result, network 
mauagement functions that could once be handled by a small group of people, l eased on intuition and 
experimentation, must now be supported l»y effective traffic-engineering tools that unite confi.yiuration and 
usage information from a variety of s^-^urces. 

♦ Traifir variability: Internet traffic is complex. Offer^i loads l»etweeu s^^mrce-d^ti nation pairs are typ- 
ically imknown. The distribution of IP traffic often ftuctuat*^ widely across time. This introduces sig- 
nificant complexity to network traffic engineering, without quieting customer demands for predictable 
communication perfomiance. Detecting and diagnosing shifts in user demands, an«T the performance im- 
plications, requires continual vigilance in network operations. Effe^^tive traffic-engineering tools should 
support, rapid identification of potential performance probleius and a flex-ible en\Hronment for experiment- 
ing ^nth possible .*;oliitions. 

The complexity of managing an ISP network stems from some of the same fundamental is^iues resjwnsible 
for the t^Mcrefifi and growth of tlie Internet. IP networks forward datagTams. and lack the Iiasic notion of a 
call or a virtual-circuit which provides the basic h^-vok for measurements and provisioning in circuit s^ntched. 
Frame Relay, anrl ATM networks. As a r^ult. load, relialiility, and performance are harder to characterise and 
track. In addition, IP networks self-configiire. via a set of interrelated . djniamic intra-domain and inter-<lomain 
proto<"ols. guided by the local configurations of network elements. Thus, rlatabas^ that are meant to hold 
authoritative descriptions of the network can diverge from the network reality, and small changes in one part 
of the network can have a significant impact on the flow of traffic. 

1.2 NetScope Tool Overview 

Managing the performance of a large TSP Tiackbone network rerjuir^s advanced tools to. support operations, 
engineering, and design. Section 2 presents l-»ackground material on TP backlxime networks and routing protocols. 
The key comp<-nients of the NetScope toolkit are shown in Figure I : 

♦ Configuration: The configuration module e\-t.ra<Tf.s a network- viHde view that includes the top^Mogj\ link 
capacities, customer adrlress. peering connections, route configuration, and layer-two conne<-ti\nty. This 
information can be extracted from the configuration files of each router, as well as the IP forwarding tables 
and inter-domain routing information, in an operational TSP network. 



♦ Measiir^^meiit: The Traffio d^maTi<1s are determine,-! based on detailed measnrements at the f>eriphery 
of the network, wliere rraffi*-^ enters and leaves the h.a<-khtone. These nieasnremeni<^ can l»e atiUT^iiated 
to the level that permits efficient and amirare modeling of inter-domain and intra-domain ron ting. The 
measnrement modnle assonates the traffic demands with the appropriate ronters and links, drawing on 
information from the configuration modnle. 

♦ Data model: A novel featnre of KetS«^ope is that it romKines diverse network ronfignratinn information 
with diverse network mea.snrements in a joint data mo^leh Attril nites of ronters. links, and traffic demands 
are derived from the configuration and measurement modules. To facilitate experimentation wHth new- 
network designs and projected trafiic demands, the tool uses simple flat files as input to the data model.- 

♦ Routing model: The routing module combines the network topology and traffic demands by modeling 
how traffic would travel through the network, leased on the intra-domain and inter-domain routing proto- 
cols. The model captures the sele<-tion of shortest paths to multi-homed customers and peers, the splitting 
of traflfic across multiple shortest^path routes, and the multiplexing of layer- three TP links on layer-two 
trunks. 

♦ Visualization: The visualization module displays the network s layer-three and layer-two connectivity, 
and provides convenient j^ccps^ to sele^^e,-] network configuration information and traffic statistics. The 
module supports coloring and sizing of routers and links l eased on a variety of statisti<-s. and computes 
histogTams. tables, and time-series plots. Tu addition, the environment allows changes to the network 
configuration to support '•what-if" experiments. 

The dotted line in Figure 1 is very significant - the network toix>logy and traffic statistics could derive from a 
wide variety of sources. For example, the topologj^ could l ^e extracted from configuration files and forwarding 
tables, tracked in real-time ly monitoring routing protocol traffic, or projected l-»ased on a proposed network 
design. Similarly, the traffic data could derive from network measurements, customer sul »scriptions. or proje<?t,ed 
demands. Se«^tion 3 descril->es the data model and how it can be popnlateii from configuration and measurement 
information. The routing and visualization modules are des^^ribed in Se^rtioi} 4 and Section 0. respectively. 

\etS<"ope is able to track change while maintaining focus on the network featur^^ of interest to the network 
architechs and operators. All aspects of the network infrastructure can and must evolve. \ew routers, line 
cards, and versions of the element control s<->ftware are regularly intr^-xluceib Ttasic archite,-tural chang*^ (such 
as the introduction of MPT/S) occur less frec^uently. Higher level modules of the software (above the line in 
Figure I ) can and must evolve to accommodate basic architectural changes. The lower level mo«lules of the 
software (below the line in Figure 1). responsil-.le for par.'^ing the raw data asscH-iatetl with network elements 
and l-»uilding higher level al-^^tractions. are designed for simplidty and extensil->ility to accommodate frecpieut. 
incremental change in the network. Extending these modules to cope with new network features as they l iecome 
of interest to network architects is a much ea.sier task than building and maintaining a tool that aims to handle 
all possible com l'»i nations of features and policies. 



Figure 2: Logi<^al viw of an TP 1>ar.kl »oTi^^ network 



Tn cl«=sigiiiiig KfttSrope, we have introdiioed partonilrir ways of represeTiting the network topologA'. inrra- 
domain and inter-domain routing, and the offered traffic. The work desoribeil in this paper fo<inses on thriving 
th^^e models, an«| defining their basic parameters. The t^-x^l does not at present support real-rime updates 
to configuration data. Such changes do not o<-cur very frequently, and hence it is reason ahle to expect some 
amount of stai n lity of the network in between failures and re<''onfiguraTions. Tu fact. XerS^^ojv^ is designed to 
study precisely these kinds of events l >y evaluating how they would afl:Vt the flow of traflRc in an TSF l 'ack>>one. 
Unlike the network model, Internet traffic does fluctuate on a variety of time scales, with important implications 
for traffic engineering. ([Congestion control mechanisms, such a<; TCV. introiiuce varialnlitj- on the small time 
scale of se«-onds [I], while variation in user deniands introduces Kurstiness on multiple time scal^ [2]. However, 
on time s^^ales l->eyond a few hours, fundamental shifts in user demands and changes in the network tojKilogy 
introduce variability >>eyond statistical fluctuations. Ofiered load typically changes with the time of the day. or 
day of the week, or in response to a network failure or re^-onfit^uratiou. Tn the traffic ent/ineering example in 
this paper, we focus on this larger rime scale. 

2 Internet Backbone Networks 

The Internet is dividetl into a colle«?tion of autonomous systems (ASs). Routing through the Internet depends 
on protocols for routing l->etween ASs, called Kx-terior ([Gateway Protocols [3,4]. and protocols for routing within 
individual ASs, called Interior ([^ateway Protocols [-3]. Each AS is managed l-»y an Internet Service Provider (ISP), 
who ojv^rates a l^ackltoue network that conne<?ts to custoniers and other service providers. After des<".ril"nnsi; the 
architecture of ISP l*tackl M->ue networks, we discuss intra-domain and inter-domain routina;. 

2.1 Backbone Architecture 

ISP I>ackl-tone networks consist of a collection of IP routers and l->i-directioual layer-three links, represented as 
nodes and edges iu Figiire 2. Arcff^.^ links coune<^t diret^.ly to customers. For example, an access link could 
connect to a mcniem bank for rlial-up users, a weMiosting complex, or a part.icular business or university 
campus. MnJti-homfil customers have two or more access links for higher capacity, load l->alancing. or fault 
tolerance. Fffring links connect to neighlx>ring service providers. A peering link could connect to a public 
Internet exchange point, or directly to a private peer or transit provider. An ISP often has multiple peering 
links to each neighlx>ring provider. tjj>ically in diflerent ge«->graphic locations. Bnrhhone links connect routers 
in.^iide the ISP backlx>ne. Routers rely on input, via a configuration file. aI»out the state of their hardware, 
the partitioning of res<">urces (such as buffers), and which policies to apply to the forwarding de^-isions. A link 
is configurecl by entering interface definitions on all routers that are part of the link. The ISP has complete 
control over the configuration and oj^^ration of backlmne links, by \nrtue of controlling the adjacent routers. 
([Configuration of access and peering links depends on interatrhion with the customers and the peers, respei-tively. 

Each router terminates a mixture of access, peering, and l-»ackbone links. To simplify the discussion, we 
assume that all acces^^ links terminate at Access Routers (.ARs) and all peering links terminate at Internet 
Gateway Router (IGRs). and that all remaining routers are Backbone Routers (BRs) that only terminate 
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harkl-M-m^^ links. Tn an operRtional n^^twork. this split, in fnnrt.ionality simplifi*^ the. r^nirenients for ^arh Toiit*^r. 
VoT t^xampl^, an A R should provide high port, dt^nsit.y to *^onne<-t to a large niiml-»er of oiistom*^rs wnih various 
aropss sp^Hs and tf^^hnologies. On the oth^r hand, a BR shonld provide high parket-forwarding perform an<"e. 
Finally, isolating peer traffir to a small set of TGRs simplifies the management of inter-domain ronting polid*^. 

TSP barkVones rnn over an iinderljSng fanlity network. This introdnoes multiple layers of ronne*-t.ivity and 
capacity, which has implications for traffir. engineering an.l reliability. The layer-three link between two adjacent 
TP ronters often corresponds to dedicated capacity at the facility level. This abstraction holds, for example, 
for packet-over-SOXKT links. However, some networking terminologies, snch as FDDT and ATM, introduce an 
intermediate switching fabric at layer two. For example, a single layer- three link may correspond to a permanent 
\-irtnal circuit (PVC!) that traverses one or more ATM links via switches. Tn fact, multiple layer- three links may 
share the capacity of a single layer-two link. AVe refer to these layer-two links as trnnk.-i. and the layer- two 
switches as ihiurfs. The traffic on a trnnk consists of the total load imparted T«y each of the layer- three links 
that traverses that trnnk. Similarly, a trnnk (or device) failure causes the failure of each of the layer-three links 
that travels over that trunk (or device). 

2.2 Routing Protocols 

Within an AS. routing is determine«1 hy interior gateway protocols such as static routing, OSPF. TS-TS. and RTP. 
Static routes are configured manually in the router. An TSP typically uses static routes to direct traffic toward 
customers who have a fixed set of TP addresses and do not participate in an inter-domain routing protocol. 
Routing prot enrols such as OSPF and TS-TS are more advanceii in the sense that routers exchange link-state 
information an«-| forvi arrl packets along short.est paths. V>as^^ on the sum of the link weights. Tj-pically. cu.stomers 
and peers do not participate directly in these protocols with the TSP. As such. OSPF typically operates only 
over the backbone links. Since OSPF uses flooding to exchange link-state information, the proto<^ol does not 
scale to large TSP backlx>nes. Therefore, TSPs topically introduce a routing hierarchy by dividing the backl-.o'ne 
into multiple OSPF nm'f<t, each running a separate instance of the protocol. 

Routing l-.etween ASs is controlled hy an ext.erior gateway protocol. T1<]tP. rt(;;P is a path- vet-tor proto^-^ol 
that rlistributes routing information T»etween routers belonging to different autonomous systems. These routers 
are rS(.;p peers that advertise and withdraw routing information aliout part.icular network addres.ses. A network 
address represents a set of contiguous TP addresses by a 32-bit number and a prefix (mask) length: for example^ 
the network address 1 :?>5. 207. 11 9.0/24 has a 24-T»it mask that specifies a l.lock of 20G TP addresses. A route 
advertisement includ^^ a list of the ASs in the path to a part.icular network address, along with other path 
attributes. For example. 

9.2.0.0/lG 192.205.31 .30 0 1740 1G73 H377 IG75 i 

des<-:ribes a route to network address 9.2.0.0/ 1 G going through a s^uence of autonomous systems, start.ing ^-ith 
the peer AS 1740. The path has a metric of 0/ and I92.20o.3 1 .30 is th^^ h-^opback address of the associated 
TGR. Upon re«-eiving a BGP advertisement, an TSP can apply Icx^al policies to decide whether to use^ the route. 
These decisions can l->e l-»as«^ on a variety of factors, .-^uch as the AS path length and Icn^al pref*^rences for 
particular dowTistream providers. After de<-iding to use a route, the TSP must also decide whether or not to 
forward the advert.isement to neighl>oring .ASs (after adding itself to the AS path). Limiting the distribution of 
advertisements allows the TSP to influence what traffic enters the network, and where. 

intimately, the router determines the path along which to forR-'ard an individual packet from the interaction 
of the various routing protocols. For example, the static routes specifie»-| for a customer on a particular router 
indicate which r^rcf^ link(s) shoulrl ultimately carry this traffic. Other routers in the backbone must learn that 
they should route traffic destined for these customer address*^ to the corT*^pondin.i>; access liuk(s). Similarly, 
the routers need to learn which peering link(s) should be used to reach each network address in the rest of the 
Tnternet. AVithin an TSP ba*-kT»one. this information is typically distributeti via TRGP (internal T^(!tP) or the 
intra-domain routing protocol. By comlouing this information with the shortest paths computed by OSPF or 
TS-TS, each router can determine the appropriate outgoing link(s) for each network addr^s. The mapping from 
network address to outgoing liuk(s) is stored in a forwarding table. For example, the *^ntry 

1 35.207 .0.0/ J G 1 2 . 1 2G .223 . 1 94 Seri al 2/0 /0 :2G 

corresponds to a link using card Serial2/0/0:2G. with next hop 1 2.1 2G. 223. 194 to forward packets toward the 
network address 13-3.207.0.0/1 G. When a packet arrives, the router performs a longest prefix match on the 
destination address to find the appropriate forwarding- table entry for the packet. Then, the router forwards the 



paokftt. to the appropriatf^ oiiTgoino; link; in ^om<^ cases, thf^. forwarrling ftntry has mnlt.ipl*^ outgoing links and 
the router can pick a single link from this set. The nexT-hnp ronter repeat-^ the process, forwarding the packet 
closer to its destination. 

3 Data Model 

Traffic engineering requires a network- wide view of the underlying topology- and an estimate of the offer^ traffic 
load. This sei-tion presents a data model that allows ns to coml«ine the net.Avork Topology with traffic statist.!*^ in 
one data strnctnre. We l .nefly dis'^uss onr approach to extracting this information from an operational network. 

3.1 Topology Model 

Onr m^xlel of TSP hackl ^onps inch ides ol j^^i^ for ronters. layer-three links; devices, and trunks; the latter t.Avo 
were defined in Section 2.1, The ronters and layer-three links are conne<-ted in a topology*, as .shown in Figure 2. 
Layer- three links have several attriliiites that play an important role in traffic engineering. Each nnidi recti on al 
link inclndes general information ahont the ronter originating the link, the name of the ronter card, the TP 
address of the interface, a teyt.nal description of its pnrp->se. its capacity, and its OSPF weight. The latter two 
are espei?ially impt->rt.ant for the ronting model. Some attributes are asso^^ate.i with lx>th dire^-tions of a link. 
For example, each hidire^-tional link can he classifie<i as an ^cref^, ]-iacklx>ne. or peering link. Backlx>ne links 
also belong to a parricnlar OSPF area, which mnst h»e the same for both nnidirectional links. Peering links are 
a.sso^".iate.-i w\i}\ a parricnlar BGP peer, identifiei-i by its AS nnmber and annotated by the TP address of the 
fJGP peer in the rennote domain. 

Ronter attriT.ntes inchide the ronter name, the h-K>pback TP addr^^^^ of the ronter. the type of the ronter 
(AR. T^R. TGR). and the ge<->graphic location of the ronter in temis of city anr! latitnde/longitnde. Tn addition, 
each ronter inclndes information abont which links it originate. The device attribntes inclnde the name and 
location of the device, and a list of trnnks that originate at the device. Trnnks describe the connectivity >>etween 
ronters and devices, and inclnde the information abont which links traverse a given trnnk. Gonseqnently. links 
have an additional attribnte that list^ the trnnks they traverse (if any), as well as the ronters on either end of 
the link. 

The model is verj' general and its ol «je<-^ts can l ^e popnlated in a nnmlw of different ways, snch as modifying 
an existing data model, constrncting an artificial network, or extracting the information from the real net^vork. 
Onr approach is to extract the information from ronter confignration files. The top'^log^- model is popnlatp<T by 
netdb, a network confignration debngger and database that also snpports interactive qneries and consistency 
che<-!k-ing [0]. \etdb prcM^ess^^ the confignration files in two .steps. The first step resolves information within a 
single file snch as interfaces, their TP addresses, cnstomer names; link .^i>eed. OSPF weight, and OSPF area. The 
second .<;tep nnites infomiation from mnltiple ronters into a single topologj* nsing TP address information. For 
example a link is identifie«1 via an TP prefix. Tf it has two or more interface entries, we classify it as a T>ackbone 
link. A link with only one TP address entry is either an ^cc*^p> or peering link. For a peering link, the as.*=;ociat*=rl 
interface participates in a TlGP peering session to a known peer. Tf a link is not a peering link it is an access 
link. 

3.2 Traffic Demands 

Fftective traffic engineering reqnires not jnst a view of the topology T>nt also an accnrate estimate of the offered 
load between varions points in the backl'»one. These estimates conld T»e derived from cnstomer snTiScriptions. 
traffic proje«Ttions. or actual measurements. Tdeally. we would like to ilefiiie a rieninmi in terms of the volume of 
load between two edge links (e.g.. from an access link to a peering link). TTowever. many cn.«^tomers connect to 
the backlM">ne via mnltiple acc^>>s links, and many external addres.ses are reachaT>le \-ia multiple peering links. 
The traffic destinet-l for a cnstomer conld flow through a different access link depending on the configniration of 
intra-domain ronting. TTence. traflRc from the external Tnternet to a customer should be modeled as a demand 
from a peering link to a sft of access links. Similarly, the traffic introduced hy a customer should T»e nnxleled as 
a demand from an Pirr€^< link to a set of peering links. A set of peering links (access links) can he. represented 
by ^ logical node Xi (>V). as shovni in Figure 3. (We s>-nonymonsly n.se A'; (Yi) to refer to either the logical 
nodes or to the set of links that these nodes represent.) Tn this paper, we fo^^iis on demands from a peer link to 
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Figiirf^ -?>: X^tworlf tofw^logj' with logiral nodes to/from f^ets of e«ige links 

a set of arress links, or from an f^rcefis. link to a set of peering links, rather than trafli*- hetween arress links or 
l ietween peering links. 

Determining the traffir demands in an operational network reqnires identifying the sets of links (.Y,- and Yi) 
and the asso<^ateii trafHr: volnmes. The sets of access links oan >»e determined l->ased on the forwnnfing tahle 
at each arress ronter. Karh table entry indicates a customer prefix (TP address and mask length ) and the «-ard 
name of the outgoing link. The card name also appears in the ronter confiy;nration file, allowing the prefix to 
l-»e associated with the appropriate link. By repeating this process across all of the ARs. we can determine the 
set of ^rr.e?s links associated vinth each customer prefix. Similarly, each external prefix can h»e associated with 
a set of peering links A';, based on information in the H(]tP routing tables. Each entry in the BGP routing tahle 
include a prefix, an AS path, and an TGR loopback addrt=ss. After identif\nng the T('4R, it is possible to identify 
the appropriate peering link(s) at that router base.-] on the nex-t-hop AS number. The loopback address of each 
TGR and the next-hop AS numl^er of each peering link can l >e extracted from the router configuration file. 

Finally, we asso<^ate each demand with a volume of traffic. Tn our work on traffic engineering, we initially 
focus on mftisnm^ traffic demands, rather than suV»scribed or proje<^te<i loads. Tn particular, we consider flow- 
level measurement^; at the network edge, where traffic enters or leaves the network. A flow consists of a set 
of packet-^ that match in all of the main TP and T^'P/l'DP header fields, such as source and destination TP 
addresses, protocol, port nnml>ers, and ty-pe-of-ser\nce h\\s, and arrive close together in time. Each measurement 
record includes information al M->nt the traffic end-p<->ints. TP and TCIP/UDP header fields, the nnml^er of packets 
and T^ytes in the flow, and the start and finish time of the flow; such information is available, for example, 
from C^iscos Netflow feature [7]. The source and destination TP addresses of the flow can Vie associated with 
the appropriate prefix . and matched to the corresponding set Xi or Yi. d'onsequently we are aV.le to compute 
comprehensive infoniiation aT>out traffic demands hy aggregating the flow level information to the level of A',- 
and Yj, 

4 Roiitiug 

Another key feature of NetS^'^ope is that it com>»ines the netw-ork model and the traffic measurements with an 
accurate mcHfel of path selection. Spenfically, \etS<"opes routing mo«fule determines the path(s) chosen by 
OSPF for each traffic demand, antf the load imparted on each link as the traffic flows through the network. The 
routing module captures the sele«-tion of shortest paths to/from mnlti-homeiH customers an< | peers, the splitting 
of traffic across multiple short,est-path routes, and the multiplexSng of layer-three links over layer-two trunks. 
These capa>>ilities in KetS<"ope allow a user to explore the impact of changes in the traffic demands or in the 
underlying network topologA'. 



FigiiTfi 4: 'TVaffi*^ :^p1iUing arro^^s multiple short.»^t paths 

4.1 NetScope Path Selection to approximate OSPF Routing 

The OSPF prororol defiTle^; how routers witliin an area exdiange link-state information and rompnre shortest 
paths l-.ased on the snm of the linlc weight-^. The link weights are sTarir. and are typically ronfignre.-! hased on 
the link capacity, physical distance, and some notion of the expected trafHc load. The chosen paths do not 
change nnless a link or router failure CM'^cnrs. or the OSPF parameters are reconfigured. These are rare evented, 
particularly for the >»ack hone links that participate in the routing protocol. As such. NetS*"ope considers a 
single instance of the network topolog>* and OSPF configuration, and does not simulate the details of the OSPF 
prot(>"ol. such as the flooding of link-state advert.isemeni':; or the exchan,!>;e of "'hello'" messages. Performing the 
path-selei?tiou computation inside the t^-v-^l. rather than using the forwarding tal lies or traceroute resu1t<; dire<-tly. 
facilitates ex-perimentation with alternate OSPF configurations and diflereut topolo,^ies. 

AVhen all of the hackln'^ne links reside in a sin,y;le OSPF area, path sele^^.ion simply involves computing the 
shortest paths l ^etween each pair of routers, l eased on the link weights. Tn a hierarchical network, traffic >»etween 
two routers in the same area follows a shortest path within the area, even if the network has a shorter path 
that involves links in other areas. Wlien traffic mu.-^t travel l-^^tween routers in <1iflrerent areas, the path dej^ends 
on how much information each area has ahout \fs neighbors. Clurrently. our routing m<-xlule assumes that the 
network does not summarize routing information at area lx>undaries. Tn the aVisence of route summarization, 
each border router report^ the cost of the short.est path(s) to each of the other routers in the area, and the 
traflfic between routers in different areas simi')ly follows a shortest path without regard to the area boundaries. 
The routes are computed using Dijkstra s shortest-path-first al.<i;orithm. 

4.2 OSPF Tie-Breaking 

Path selection l->e<"omes more complex when there are mnJtipJf shortest paths l»etween a pair of routers. Such 
ties arise very naturally when the network tof>ologj* has parallel hnks Itetween adjacent routers for additional' 
capacity. Ties also surface when many of the links in the network have similar weight^. This is sometimes done 
intentionally to increase the effective capacity b»etween two end-poiuis. The presence of multiple shortest paths 
allows for load-balancing of the traflfic Vietween the two end points. This is achieve^"! >>y allowing the TP forw-arding 
tal ile to have multiple outgoing links associated with a single destination prefix. Rather than alternating between 
these links at the packet level , routers typically attempt to forward packet^; for the same source-destination pair 
along a single path; this reduces the likelihoo«T that packets from the same T(!IP connection arrive out-of-order at 
the receiver. Load-l'>alancing is typically achieved by perfonning a hash function on the source and destination 
TP addresses of each pactket. The value of the hash function determines which outgoing link should carry the 
packet. 

Tn theory, the details of the tie-breaknng function could l-»e modeled in the KetScope tool. TTowever. this 
would significantly complicate the path-sele«"tion computation, and would require computing traflfic demands at 
a significantly finer level of gTanularity. Tn addition, the details of the hashing function, and how the outputs of 
the hash function map to particular outgoing links are not spe«-ified l->y the OSPF proto^^ol and, as such, depend 
on the vendor's implementation. Fortunately, these details are usually not important. The hash functio7i is 
designed to support an even splitt.ing of the traflfic across the multiple outgoing links, especially for backlx>ne 
links that carry a diverse mixture of traffic with different source and destination addresses. As such, our routing 
model splits traffic evenly across each of the outgoing links along a shortest path. For example, if a router has 
four outgoing links on shortest paths, each link would carry *2o% of the traffic. The divi.^ion of traffic is recursive, 
with the downstream routers dividing the traffic across each of their outgoing links, as shown in Figure 4. 



Figure 5: The Tiink Panel, displaying the atrribntes of a l-.a*^Vlv>ne link. 
4.3 Multi-homed Customers/Peers 

As disr.ii.ssed in Sei^.ion 3.2, traffic from a mstomer is represented as a demand from an acress link to a set of 
peering links (A';). Similarly, the traftic to a rnstomer is represented as a demand from a peering link (where 
the traffir. was measnred) to r set of aroess links (Yi), The choice of a parti cnlar r>eering link (access link) from 
the set Xi [Yi] depends on OSPF routing. A set of links can he represented as a logical no<1e. as .^hown in 
Figure 3. Traffic travels to the cli->sest link, along a short.est path. The chosen link, as well as the ch^-^sen path, 
depends on the link weights. We model the sele^^tion of the short.est path to the closest node l iy assuming that 
the logical links all have the same OSPF weight (e.g.. a nominal weight of J ). Under this assnmption. a shortest 
path to a logical node Xi [Y^] travels throngh the appropriate peering link (access link). 

Tn the end, the ronting nvxlnle operates on a set of demands, each traveling from one peering link (access 
link) to a set of access links (peering links). The module computes the set of shortest-path rontes ha.-^^ed on 
the topology and the OSPF confignration, and determines how the demand splits acr<->ss the multiple paths. 
Repeating this process for each demand resnlts in an estimate of the loa<i iinparted on each link. Then, the 
routing modnle determines the load on each t.rnnk (layer- two link) by summing across the associated (layer- 
three) links. The generality of the ronting model facilitates experiments \\nth alternate topologies and OSPF 
configurations, as illu.strated in the next section. XetScope also supports e\-x>erimentation w]th the.B(]TP policies 
for outlx>nnd traffic, by changing the sets of peering links associated with external network addresses, 

5 Visualization 

The XetScope visualization environment provides many ways to explore the data asso«^ated with an TP backbone 
network, and the ability to perform '^hat-if " experiments. This .section gives a brief overview of the visualization 
en\nronment. followed by a demonstration of using the tool. The demonstration addresses the traffic-engineering 
task of reducing the load on the network *s most utilized link, using an artificially constnictei-| topology and set 
of traffic demands. 

5.1 Objects 

The NetS^^ope data model is (decomposed into a set of ol-»jei-t^. e.g., oh.jects representing each router and each 
link. We associate a list of attributes with each object, as discussed in Sei^tion 3. 1 . The visualization environment 
provides a way to examine each obje«-t and see all of it^^^ attributes. For example, Figure 'j shows the T.ink Panel. 
The attriliutes <-|isplayed are for the link underlinetl in the upper left list, namely, the interface POS3/0/0 from 
a router in Dallas (which terminates in a router in T/*'>s Angeles). This link is implemented vna packet-over- 
SONRT te«-hnologA' and therefore has only one associated physical link (pi ink), itself. Ket Scope allows for easy 
navigation l-»etween the obje^^ts. For example, starting at a link it is straightforward to find the routers where 
this link terminates. Starting at a router it is easy to find the links that terminate there. 



5.2 Statistics 



XetS^opf; maintains statistical information assoriat^d with ohj»=s?ts. Karh statistic is simply a va1iit=^ for •=iarh 
ol-tj*=H^.t of som*=j t}7>^. For ^xampl^. a link utilisation statistic is a p^rci^ntag^^ ass^-^ciat^d w\xh e^adi link. Th*=^rv^ is 
no r^trirtion on liow many statistic^ ran l »e associated with an ol-.j*^.t tj^pe. For example, it is possible to keep 
link ntili nation on an honrly basis for a full week. Statisti*^ can be stati<^ (read from a hie. or computed once) or 
dynamic (antomatically recomputed as needed). XetScope has many ways of rlisplaying statistics. For objects 
that have graphical representations such as links or nodes. NetS^'^ope can make the sise or color of the ol ject 
>.e prop<->rt.ional to the value of the statistic, providing a \nsnal representation of the statistic. The first step 
of oiir traffic engineerina; task is to visualise the utilisation. Figure 0 shows such a screendump from \et,S^ope 
operating on our artificially constructed network topolog>^ Each nocle corresponds to a router anci each edge 
corresponds to a backbone link. The thickness and the color of the edges corresponds to the utilization on that 
link. The hnks vinth low utilisation are thin and yellow, vihile those with high utilisation are thick an-l red. 

The smaller window in the figure is the T.ink Statisti^-s window. This window .lisplays information about 
statistics, and allows the user to visualise statisti*^ in many ways. From this wnndow, the user can spe*?ify which 
statistic (if any) should >>e used to determine the size or c«->lor of the obje«?tj^. The statisti<-s window also displays 
summary statistics for each link statistic. Under the columns labele^i Min, Max, and Ave. we see the minimum, 
maximum, and mean value (over all links) for each displayed statistic, e.g.. the ave is 8.83%. 

XetScope has the notion of a current ol-tject for ea<"h type. An ol>je<^. l^ecomes the current object, either 
when it is selected or when the mouse is moved on top of it. The column labeled Chirrent shows the utilization 
value for that link, vtnth the name of the current link displayed below the menu-bar. Tn the statisti*^ window in 
Figure G, we see that the current link is the interface POS?>/0/0 on a backhone router in Dallas, and that the 
utilization on that link is 29.27%. Also from this window*, the user can produce other \-iews of statistics, such 
as histograms (showing how many links have a utilisation that falls into a certain range) or tables. Tt is also 
possil'>le to correlate different statistics for the same oh'ject type. This can h»e done either \na a scatter plot, or 
by. for example, using the color of the lines to \-isualize highly utilised links and using thickness of the lines to 
\nsualise links with high delay. 

5.3 Locating Heavily-Loaded Links 

To finish the first step of our example traffic engineering task, the \etScope user needs to i«-|entify the most 
heavily-loaded link. KetScope has powerftil search tools that allow the user to perform complex queries on all 
obje^-ts via a Find panel, as showm in the example in Figure G. ([riven that the max-imum utilization for our 
example is G7.09%., the user asked the xo*-A to locate all links w\fM a utilisation value alcove G0%. By passing 
the mouse over each link listeii here one can easily find the link with the highest utilisation. (There are also 
other ways to find this link, such as a histogram or ta>»le.) Tn this case, we -find that the link with the highest 
utilisation is a cross-country link from Washington, D.([^ to San Francisco. 

The queries allowed in the Find panel are very general and can refer to arbitrary attributes and statisti*-^;. 
After 1o«"ating an interesting subset of data. \etScope allows the user to restrict the current view to only that 
sul'tset. For example, if the u.ser was only interested in the backbone links, the user could find that set of 
links and then make that the active .set by clicking on the '^'Make Active-* button. ([Ihanging the active set 
automatically causes .*^veral changfi^. The display changes to show only those links that are now active. The 
minimum, maximum, and average are re*-omputed over the new active set. Finally, the coloring and si sing of 
links (if turned on) will change to refle*^. the new range of values. 

5.4 Traffic on a Link 

<Jontinuing \^Hth the traffic engineering example, we next nee^l to identify which traffic demands contribute to 
the load of the highly utilised link. The link carries traffic for a collection of different demands. KetS^^ope's data 
and routing m<xlels allow us to see the source and sink tree of all traffic demands usin.o; the link. Kac]i object has 
a menu ass<:>ciated with it. with operations that apply to that obje«Tt. This menu includes some general object 
operations such as "^get information.' Sele«-t." ''scM-ym." and operations specific to the oV'ject type. The link- 
menu, for example, includes the option for finding all traffic demands with short.est path routes that include this 
link. This query is executed via the demand Find Panel. By using the ''Make Active" button in this window, 
we restrict XetS^^'opeV \Tew to this set of demands. The statisti<"s and their graphical visualisation (e.g., the 
colors of the links) will hte adjuste>-i automatically. 
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Fi^^nire G: (L^ft) Link iitili nation display^ as siz^ and color of links. The small^^r window is the Link Statistics 
window. (Right) Finding all links wirh iitili^ration a>ove G0%. 




Figure 7: (Left) The ntilization cansed hy a snl-«>et of the demands, namely all the demands that impose some 
traffic onto the most heavily loaded link. (Right) The ronting of a demand from' AVashin,ti;ton DM-, to Seattle. 

To fcH^iis onr attention on the chosen demands, we activate only the links that carry one or more of these 
demands. The resnlting \-iew is shown in Fignre 7. At first glance, it may seem confnsing that the resnlring 
graph is not a tree. TTowever. OSPF tie-breaking intrcwlnces mnltiple paths for each demand, hence we get 
a i-|ire<-t.ed acyclic graph instead of a tree. AMiar appears to l-»e a cycle in the part of the topolog>^ sho^^ in 
Fignire 7 are directed links pointing to either San Francisco or from Washington. D.(]!. To iHnstrate the impact 
of OSPF tie-hreak-ing on the rente selection. Figure 7 also shows how one such demand is ronted throngh the 
netMwk. This ronte from Washington, T).C. to Seattle uses three paths throngh the network, one of them nsing 
the hea\*ily ntilize.-] link from Washington, T).C, to San Francis^-o followed hy the link from San Francisco to 
Seattle. The other two rontes nse the link from Washington, T).C. r'hica.t^o and then the two parallel links 
lietween (I Chicago and Seattle. 

5.5 Changing Routes 

The ohvions approach to alleviate tlie congestion on the highest, ntilized link, other than increa.-^ing the capacity 
of the link, is to increase the OSPF weight of the link in order to change the ronting. This will tend to move 
traffic from that link to other links. \etScope allows the nser to modify OSPF weights. Upon changing the 
OSPF weights, XetScope recalcnlatft^ all rontes for all active traffic demands. The tool then nplates all statistics 
that are based npon the traffic, including link load and ntili oration. Tn or<1er to compare alternate settings of 
OSPF weight^, NetScope maintains two different sets of weights, one that can hie manipulated and one that 
acts as an anchor. 
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F\giire 9: D^r^^asing an OSPF wv^iglit. on a diff<^r^nt linl:. This gives a better solution. 

Figure 8 shows the network utilization l-»efore increasing the OSPF weight of the highest ntilized link anH after 
changing the weight. To simplify the visnali;^ation we nse a different coloring scheme in this and snKseqiient 
■figures. Links with low utilization (at most 30%) are green; link's with me.iiiim ntili/ration (l-»etw*een 30% 
and G0%.) yellow; links with high utilization (over G0%-) red. Before the change the most utilized link had 
a utilization of 07.59%.. Changing the OSPF weight reduces the link utilization to 00.09%-. Unfortunately, 
a link from Washington, D.(]^. to C^hicago that previously had a fairly small utilization, now has utilization 
09.70%.. Tt is possible to explain this l»y going V>ack to Figure 7. By increasing the OSPF weight on the link 
between Washington. T).C. and San Francisco, the path from Washington. D.C to San Ftandsco to Seattle is 
remo\^ from the path set for the demand from Washington, Y).C. to Seattle. The same is true for the demand 
from Washington. T).C. to Dallas. The implication is that the traffic that use«1 to flow on the link between 
Washington, D.C and San Francisco now traverses the link from Washington, D.C. to (IHiicago. This illustrates 
one of the difficulties in managing an TP network: a small loi?al change can have signiiflcant global impact. 

Based uix>n this understanding of the traffic pattern, one might want to decrease an OSPF weight on a 
different link, so that it will attract more traffic. From the alx>ve dis<"u.ssion one can identify the traflRc demand 
between Washington, T),C. atid Dallas as a good target, d'onsequently the link Iw^tween Washington. D.C. and 
St. bonis is a good target link. Figure 9 shows the result of such a change. This time we succeede* ! in de<-.reasin.*i 
the total maximum utilization to less than 01. 5%'. To decrease the utilization even further, a traffic engineer 
might want to iterate the process that we have illustrated for the now highly utilized link h»etween C-hicago and 
(^lambridge. Alternatively, an optimization tool can automate all or part of these ta.^^ks; this reciuires efficient 
algorithms for sele<Thiug OSPF weights base. "I on a given topologj" and set of traffic demands [8]. 
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6 Couchisioiis aud Future Work 



The XetScope to->lkir integrated amirate m«->dels of topolog}^ traflfir. and Tontin,i>; with a flexilMe visnalizarion 
en\-ironmenT to siipiv:>rt traffic engineering in large TSP networks. The power of the tool stei-n.-^ from the integrat*^ 
data mode] that provides a network- Avide \-iew of ronfign ration information and traffir stati.sti*'^ t^o.aierher with 
a routing mode]. The topologj- model indiides the network connectivity, link capacities, and The parameters 
controlling intra-«-lomain ronting, derived from the router confignration fil^d. Traffic demands are rompitted 
l-.ased on f^ige measnrenienrs, an*| are aggregatpd to the coarsest level that still permits accnrate modeling of 
intra-domain and inter-<1omain routing. Drawing on the traffic demands and the topology m^xlel. the ronting 
mo^liile determines the load imparted on layer-three links and layer-t^^o trunks nniier a variety of network 
configurations. These modnles. conpled with the flexible vnsiiali nation environment, en aide nsers to investigate 
alternative configurations ofl^-line withont disrupting the operational network. Tn addition . the framework readily 
supports additional modules for automated network optimization and performance deTnigging. 

As part of our ongoing work, we are evolving the tool to operate on a continuous feed of topology and 
traffic data, to track changes in the network configuration and usage patterns. Tn addition, we are analyzing 
the statistical properties of the measured traffir deuiands to determine bow the load h>etween access and peering 
links fluctuates on various time scales. This traffic analysis is critical to determining the appropriate time scale 
for exernsing control over the network, siirh as changing the couflguration of intra^lomain or inter-<-lomain 
routing. We are also investigating how the traffic demands would change after a network rei^onflguration. For 
example, flows regulatei-l l-.y congestion control me^^ianisms. such as TCP, may >ie afile to transmit packets 
more aggressively if a ronting change allevnates congestion on lx>ttlenecked links. Tn addition, we are ex7>loring 
the use of XetScope for traffic engineering an^T admission control for the mix-ture of traffic classes proposed for 
diflerentiated services. We l*»elieve that the "N'etScope approach to comhoning top<">logyj traffic, and routing can 
serve as a general underpinning for managing the performance of large TSP networks. 
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